On the Convergence of Projected-Gradient Methods with Low-Rank Projections for Smooth Convex Minimization over Trace-Norm Balls and Related Problems

Smooth convex minimization over the unit trace-norm ball is an important optimization problem in machine learning, signal processing, statistics, and other fields that underlies many tasks which one wishes to recover a low-rank matrix given certain measurements. While first-order methods for enjoy optimal convergence rates, they require worst-case compute full-rank SVD on each iteration, order Euclidean projection onto ball. These computations, however, prohibit application of such large-scale problems. A simple natural heuristic reduce computational cost approximate using only SVD. This raises question if, under what conditions, this can indeed result provable solution. In paper we show any solution center inside projected-gradient mapping admits rank at most multiplicity largest singular value gradient vector point. Moreover, radius scales with spectral gap vector. We how readily implies local (i.e., from “warm-start" initialization) standard as method accelerated methods, computations. also quantify effect “over-parameterization," i.e., computations higher rank, ball, showing it increase dramatically moderately larger rank. extend our results setting smooth regularization bounded-trace positive semidefinite matrices. Our theoretical investigation supported by concrete empirical evidence demonstrates correct projections completion task real-world datasets.